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Abstract. On a particular example we describe how to state and to solve the 
problem of harmonic analysis for groups with infinite-dimensional dual space. The 
representation theory for such groups differs in many respects from the conventional 
theory. We emphasize a remarkable connection with random point processes that 
arise in random matrix theory. The paper is an extended version of the second 
author's talk at the Congress. 



Introduction 

In this paper we would like to discuss a connection between two areas of math- 
ematics which until recently seemed to be rather distant from each other: (1) 
noncommutative harmonic analysis on groups and (2) some topics in probability 
theory related to random point processes. In order to make the paper accessible 
to readers not familiar with cither of these areas, we will explain all needed basic 
concepts. 

The purpose of harmonic analysis is to decompose natural representations of a 
given group on irreducible representations. By natural representations we mean 
those representations that are produced, in a natural way, from the group itself. 
Examples include the regular representation, which is realized in the space on 
the group, or a quasiregular representation, which is built from the action of the 
group on a homogeneous space. 

In practice, a natural representation often comes together with a distinguished 
cyclic vector. Then the decomposition into irreducibles is governed by a measure, 
which may be called the spectral measure. The spectral measure lives on the dual 
space to the group, the points of the dual being the irreducible unitary repre- 
sentations. There is a useful analogy in analysis: expanding a given function on 
eigenfunctions of a self-adjoint operator. Here the spectrum of the operator is a 
counterpart of the dual space. 

If our distinguished vector lies in the Hilbert space of the representation, then 
the spectral measure has finite mass and can be normalized to be a probability 
measure.^ 

Now let us turn to random point processes (or random point fields), which form 
a special class of stochastic processes. In general, a stochastic process is a family 

^It may well happen that the distinguished vector belongs to an extension of the Hilbert space 
(just as in analysis, one may well be interested in expanding a function which is not square 
integrable). For instance, in the case of the regular representation of a Lie group one usually takes 
the delta function at the unity of the group, which is not an element of . In such a situation 
the spectral measure is infinite. However, we shall deal with finite spectral measures only. 
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of random variables, while a point process (or random point field) is a random 
point configuration. By a (nonrandom) point configuration we mean an unordered 
collection of points in a locally compact space X. This collection may be finite or 
countably infinite, but it cannot have accumulation points in X. To define a point 
process on X, we have to specify a probability measure on Conf(X), the set of all 
point configurations. 

One classical example is the Poisson process, which is employed in a lot of prob- 
abilistic models and constructions. Another important example (or rather a class 
of examples) comes from random matrix theory. Given a probability measure on 
a space of N x N matrices, we pass to the matrix eigenvalues and thus obtain a 
random A^-point configuration. In a suitable scaling limit transition (as N — » oo), 
it turns into a point process living on infinite point configurations. 

As long as we arc dealing with "conventional" groups (finite groups, compact 
groups, real or p-adic reductive groups, etc.), representation theory seems to have 
nothing in common with point processes. However, the situation drastically changes 
when we turn to "big" groups whose irreducible representations depend on infinitely 
many parameters. Two basic examples are the infinite symmetric group ^(oo) 
and the infinite-dimensional unitary group U{oo), which are defined as unions of 
ascending chains of finite or compact groups 

S{1) C S{2) c 3(3) C . . . , C/(l) C U{2) c [/(3) C . . . , 

respectively. It turns out that for such groups, the clue to the problem of harmonic 
analysis can be found in the theory of point processes. 

The idea is to convert any infinite collection of parameters, which corresponds to 
an irreducible representation, to a point configuration. Then the spectral mcasm-c 
defines a point process, and one may try to describe this process (hence the initial 
measure) using appropriate probabilistic tools. 

This approach was first applied to the group 5(00) (see the surveys Borodin 
Olshanski [B02], Olshanski [016]). In the present paper we discuss the group U{oo), 
our exposition is mainly based on Olshanski [017] and Borodin-Olshanski [B06]. 
Notice that the point processes arising from the spectral measures do not resemble 
the Poisson process but are close to the processes of random matrix theory. 
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first author (A. B.) served as a Clay Mathematics Institute Research Fellow. He 

was also partially supported by the NSF grant DMS-0402047. The second author 
(G. O.) was supported by the CRDF grant RM1-2543-MO-03. 

1. Dual space and the problem of harmonic analysis 

Recall that a unitary representation of a group G in a Hilbert space ff is a 
homomorphism of G into the group of unitary operators in H. For instance, if G is 
a locally compact topological group then there is a natural representation generated 
by the (say, right) action of G on itself, called the regular representation. Its space 
is the space formed with respect to the Haar measure on G, and the operators 
of the representation are given by 

{R{g)f){x) = f{xg), geG, x G G, f & L\G). (1.1) 

A unitary representation is said to be irreducible if it is not a direct sum of 
other representations. Irreducible representations are elementary objects like simple 
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niodiilcs. A general unitary representation T is, in a certain sense, built from 
irreducible ones: in simplest cases T is decomposed into a direct sum of irreducibles, 
and in more sophisticated situations, direct sum is replaced by "direct integral".^ 
Two fundamental problems of unitary representation theory are: 

1. Given a group G, find all its irreducible unitary representations. 

2. For most natural representations of G (e.g., the regular representation), de- 
scribe their decomposition on irreducibles. 

The set of (equivalence classes of) irreducible unitary representations of G is 
called the dual space to G and is denoted by G. Thus, the first problem is the 
description of G. The second problem is called the problem of harmonic analysis. It 
can be viewed as a noncommutative generalization of the classical Fourier analysis. 

These two problems were extensively studied for "conventional" groups. The 
existing literature is immense, and surveying it is beyond the scope of the present 
paper. What is important for us is that both problems, with appropriate refinement, 
make sense for certain "nonconventional" groups as well. These are the groups 
of automorphisms of infinite-dimensional Riemannian symmetric spaces and also 
certain combinatorial analogs of such groups, which are built with the help of the 
infinite symmetric group. 

Results on construction and classification of irreducible representations for the 
automorphism groups and their combinatorial analogs can be found in Olshanski 
[Oil], [015], [012], [014], Pickrell [Pi2], Nessonov [Nes]. The construction of natural 
reducible representations for these groups and related questions are discussed in 
Pickrell [Pil], Kerov-Olshanski-Vershik [KOVl], [K0V2], Olshanski [017]. In the 
present paper we focus on a single group G, which is U{oo) x U{oo). The reason why 
we consider not the group U (oo) but the product of its two copies will be explained 
below. Here we would only like to note that U (oo) (or an appropriate completion 
thereof) can be viewed as an infinite-dimensional Riemannian symmetric space, 
and then U (oo) x U (oo) arises as a group of automorphisms of that space. 

2. The dual space U{N) and spherical representations of U{N) x U{N) 

In this section we briefly describe a few necessary facts about representations of 

the groups U{N). The material is classical, we present it in a form which will help 
to understand the subsequent infinite-dimensional generalization. 

For N = 1, 2, . . . let U{N) denote the group of unitary matrices of size N x 

N. This group is compact. Its irreducible representations arc parametrized by 
signatures of length N, that is, iV-tuples A = (Ai, .... Ajv) of integers such that 

Ai > • • • > Xn^ Thus, the dual space U{N) can be viewed as a countable discrete 
subset of K^. 

Let TT^ denote the irreducible representation corresponding to a signature A G 

U{N), dimTT"^ denote the dimension of the representation space, and Rn be the 
regular representation of U{N) in the Hilbert space L'^{U{N)). The decomposition 

^This claim is true under certain assumptions on the group G or on the representation T, but 
we don't want to discuss technicaUties here. Under additional (but still rather broad assumptions), 
the decomposition into irreducibles is essentially unique. 

^See, e.g., Weyl [We], Zhelobenko [Zhe], Helgason [He]. 

■* Another term for collections A is "dominant highest weights for U (AT)" . 
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of -Rjv looks as follows 

= dimTT^-TT^ 

xeu{N) 

In other words, each irreducible representation enters the regular representation 
with multiplicity equal to the dimension of this irreducible representation. This 
is a special case of a general result valid for any compact group, the Peter-Weyl 
theorem. 

We observe now that the group U (N) acts on itself both on the right and on 
the left, so that U{N) becomes a homogeneous space U{N) x U (N) / diag(U (N)) , 
where dia.g{U{N)) stands for the diagonal subgroup in U{N) x U{N). This enables 
us to extend the representation Rn to a, unitary representation Rn oi the group 
U{N) X U{N) acting in the same space L'^{U{N)), cf. (1.1): 

{RNi9i,g2)f)ix) = fig^'xg,), (ffi,52) G U{N) x U{N). 

We call Rn the biregular representation. 

In contrast to Rn, the decomposition of Rn is multiplicity free: 

^JV= (7r^0 7r^*). (2.1) 

Here tt^ stands for the conjugate representation to w^; its signature is A* = 

(— Aat, . . . , — Ai). We observe that general irreducible representations of U{N) x 

U{N) are of the form ig) tt'^, where A, /z € U{N). Representations with ji = \* 
are characterized as those possessing a spherical vector, that is, a nonzero vector 
invariant under the subgroup diag(C/(7V)). Such representations arc called spheri- 
cal. The whole subspace of diag(J7 (A'^))-invariants in tt^ ® tt^' has dimension 1, so 
that the spherical vector is defined uniquely up to a scalar factor. Therefore, the 
spherical vector is a distinguished vector in the representation space. 

Note that the homogeneous space U{N) x J7(A^)/diag(C/(iV)) is an example 
of a compact symmetric space G/K. For any such space, the associated unitary 
representation of G in L'^(G/K) is multiplicity free and its decomposition involves 
exactly the irreducible spherical representations of the pair {G,K), that is, those 
irreducible representations of G that possess a JsT-invariant vector. 

Returning to our special situation we conclude that the dual space U{N) ad- 
mits an alternative interpretation as the set of (equivalence classes of) irreducible 
spherical representations of the pair {G,K) = {U{N) x U{N),dia,g{U{N))). 

Now we shall explain how this picture transforms when U{N) is replaced by 
U{oo). 

3. The dual space f/(oo) and spherical representations of U{oo) x U{oo) 

Consider the tower of groups U{1) C U{2) c U{3) C . . . where, for each N, 
the group U{N) is identified with the subgroup in U{N + 1) formed by matrices 
g = [gij] such that gi,n+i = 5n+i,i = <5j,n+i. We define U{oo) as the union of all 
groups U{N). Equivalently, U{oo) consists of unitary matrices g = [gij] of infinite 
size, such that gij = dij for i + j large enough. 
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The conventional definition of a dual space, when applied to the group [/(oc), 
gives a huge pathological space. ^ It turns out that the situation drastically changes 
if we mimic the alternative interpretation of U{N) stated at the end of §2: 

Definition 3.1. We set U (oo) to be the space of (equivalence classes of) irreducible 
spherical unitary representations of the pair {G, K), where 

G = C/(oo) X t/(oo), K = diag{U{oo)). (3.1) 

Here "spherical" has the same meaning as above: existence of a nonzero K- 

invariant vector. Again, such a vector is then unique, within a scalar factor. Below 
M.+ C M denotes the set of nonnegative real numbers and R"^ denotes the direct 
product of countably many copies of M+. 

Theorem 3.2. The space U{oo), see Definition 3.1, can he identified with the 
subset n c R+°°+^ X X X X M+ X M+ formed by 6-tuples uj = 

(a"*", Z?"*", a~, /3~, J"*", 5~) such that 

a± = (af > > • • • > 0) e R^, 13^ = {pf > > ■ ■ ■ >(i) & R~, 
5±eR+, /3+ + /3r<l, Y.^af+I3f)<5^. 

i>l 

Thus, for any point u; e there exists an attached irreducible spherical repre- 
sentation of (G, K) which we denote by T'^. Representations T'^ enter a larger class 
of admissible representations which are studied in detail in Olshanski [015], [013]. 
In particular, we dispose of an explicit description of the representation space of 
T'^ together with the action of G in it. 

Theorem 3.2 has a long history. First of all, it should be said that the classifica- 
tion of irreducible spherical representations of (G, K) is equivalent to that of finite 
factor representations of the group C/(oo), see Olshanski [Oil], [015, §24].^ Finite 
factor representations of J7(oo) were first studied by Voiculescu [Vo]. He discovered 
(among many other things) that these representations are parametrized by the so- 
called two-sided infinite totally positive sequences of real numbers. But he did not 
know that such sequences were completely classified much earlier by Edrei [Ed]. 
This fact was pointed out later by Vershik-Kerov [VK2] and Boyer [Boy]. Thus, 
Theorem 3.2 is hidden in Edrei's paper. Note that [Ed] is a pure analytical work, 
which at first glance has nothing in common with representation theory. Another, 
very different approach to Theorem 3.2 was suggested in Vershik-Kerov [VK2] and 
further developed in Okounkov-Olshanski [OkOl]. 

Let SGN{N) c denote the set of signatures of length N, see §2. We shall 
now define a sequence of embeddings tjv : SGN(A^) — > SI such that as N ^ oo, 
the image t]v(SGN(A'')) becomes more and more dense in fl. This agrees with the 
intuitive idea that the space U (oo) should be a limit (in an appropriate sense) of 
the spaces U{N). First, we need 

®This is a general property of the so-called wild groups; U{od) is one of them. 
®About factor representations, see, e.g., Naimaxk [Na, §41.5]. In the present paper we do not 
use this concept. 
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Definition 3.3 (Vershik— Kerov [VKl]). Let /i be a Young diagram, /i' denote 
the transposed diagram, and d{fj,) denote the number of diagonal boxes in fi. We 
also regard fi as a partition = (/xi, /i2, • • • )) so that /Xj is the length of the ith row 
in fj. while is the length of the ith column. The numbers 

Oi(/z) = /ij - i+ 5, =/x--i + i, l<i<d{iJ,) 

arc called the m,odified Frobenius coordinates of fi. 

For instance, if /j. is the partition (3, 3, 1, 0, 0, ... ) then d{iJ,) = 2 and ai(/x) = 2^, 
0'2{l-i) = I5, &i(m) = 2^, &2(m) = |. The modified Frobenius coordinates are always 
positive half-integers whose sum equals |/u|, the number of boxes in fj,. 

Definition 3.4 (Embedding ln ■ SGN(iV) fl). Given a signature A G SGN(Af), 
we represent it as a couple (A"*", A~) of Young diagrams corresponding to positive 
and negative coordinates in A: 

A = (A+ > • • • > A+ > 0, . . . , > -A;- > • • • > -A]-). 

Then we assign to A a point w = i.jv(A) e O, see Theorem 3.2, as follows 

^± = 1^. i<d(A±). i<d(A±). 
' \ 0, i> d(A±) ' \ 0, i> d(A±) ' N • 



It is readily verified that lo = (a^, Z?"*", a", (5^) is indeed a point of Q. In 

particular, the inequality + < 1 follows from the evident fact that k + l < N. 

We equip Q with the topology inherited from the ambient product space M^°°"'"^. 
Then any point lj £ fl can be approached by a sequence of the form /,jv(A^^-'), 
where A*^^-' G SGN(A^), 00. Moreover, given a sequence {A'-^-'}, we have 

(.;v(AW)-a;) ^ (tt^'"^ ® tt^'"** - T^) , 

where the last arrow means the convergence of representations of the groups U (N) x 
U{N) to a representation of the group G = U{oo) x U{oo), as defined in Olshanski 
[015, §22], [012]. 

4. The problem of harmonic analysis 

Let us try to understand now what could be an analog of the decomposition 
(2.1) for the group G. From §3 wc already know the counterparts of the discrete 

set U{N) and of the representations tt^ (g) tt'^ : these are the infinite-dimensional 
space O and spherical representations T". But what is the counterpart of the 
biregular representation Rn acting in the Hilbcrt space L'^{U{N))? 

The conventional definition is not applicable to the group U{oo): one cannot 
define the space on this group, because U{oo) is not locally compact and hence 
docs not possess an invariant measure. To surpass this difficulty wc embed U (00) 
into a larger space it, which can be defined as a projective limit of the spaces U (N) 
as N ^ 00. The space il is no longer a group but it is still a G-space. That 
is, the two-sided action of U{oo) on itself can be extended to an action on the 
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space il. In cxmtrast to [/(oo), the spac;c; ii posscsscis a biinvariant finite measure, 
which should be viewed as a substitute of the nonexisting Haar measure. Moreover, 
this biinvariant measure is included into a whole family {^(^^j^gc of measures with 
good transformation properties/ Using the measures /i^"' we explicitly construct 
a family {Tz,w}z.wec of representations, which seem to be a good substitute of the 
nonexisting biregular representation. In our understanding, the T^^^'s are "natural 
representations" , and we state the problem of harmonic analysis on U{oo) as follows: 

Problem 4.1. Decompose the representations T^^w on irreducible representations. 

We skip a concrete description of the representations which can be found 

in Olshanski [017] , and only list some of their properties that are relevant for our 
discussion. Henceforth we will assume that ^{z+w) > — 1 and that z and w are not 
integers. Then, as it follows from the construction, T^^w comes with a distinguished 
unit vector ^, which is iC-invariant and cyclic. The latter property means that the 
linear span of the G-orbit of ^ is dense m H = H{Tz,w), the Hilbert space of Tz,w 
Let Hn C H he the Hilbert subspace spanned by the orbit of ^ under the subgroup 
U{N) X U{N) C G. Then carries a unitary representation of U{N) x U{N), 
which turns out to be equivalent to the biregular representation Rn of §2. Since 
{Hn} is an ascending chain of spaces whose union is dense in _ff, we sec that Tz_w 
is an inductive limit of the biregular representations Rn . At this place the reader 
might ask about the meaning of parameters z, w; the answer is that to each value 
of {z, w) there corresponds a specific tower of embeddings 

Hi = L''{U{1)) C---cHn= L\U{N)) c Hn+1 = L\U{N + 1)) c . . . . (4.1) 

There are many (even too many) ways to realize J?jv as a subrepresentation of 
Rn+1, and our construction leads to a distinguished 2-parameter family of towers 
of embeddings. 

The statement of Problem 4.1 looks rather abstract but we will gradually reduce 
it to a concrete form. The first step is to apply the following abstract claim. 

Theorem 4.2. Let T be a unitary representation of G in a Hilbert space H and as- 
sume that there exists a K -invariant cyclic vector & H (we will assume ||^|| = 1). 
Then (T, ^) is completely determined, within a natural equivalence, by a probability 

measure P on the dual space t/(oo) = 17. The decomposition ofT on irreducible rep- 
resentations is given by a multiplicity free direct integral of spherical representations 
T^ with respect to measure P. 

We call P the spectral measure of (T, ^). Note that if ^ is replaced by another 
vector S,' (z H with the same properties then P is replaced by an equivalent measure 
P' . We will not define precisely what is a "direct integral of representations" (sec, 
e.g., Naimark [Na, §41]) but only observe that Theorem 4.2 is strictly similar to a 
customary fact, the spectral theorem for a pair (^4, ^) where A stands for a self- 
adjoint operator in a Hilbert space H and ^ G is a unit cyclic vector. 

Taking into account Theorem 4.2 we replace Problem 4.1 by 

''The idea to enlarge an infinite— dimensional space in order to build measures with good trans- 
formation properties is well known. This is a standard device in measure theory on linear spaces, 
but there are not so many works where it is applied to "curved" spaces (see, however, Pick- 
rell [Pil], Neretin [Ner]). For the history of the measures ^("f we refer to Olshanski [017] and 
Borodin— Olshanski [B05]. A parallel construction for the symmetric group case is given in Kerov- 
Olshanski-Vershik [KOVl], [KOV2]. 
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Problem 4.3. Assume that z,w G C \ Z and 3R(z + w) > —1. Let ^ be the 
distinguished K -invariant cyclic unit vector provided by the construction ofT^^w, 
and let Pz,w denote the spectral measure of {Tz^w, 0> which is a probability measure 
on fl. Describe Pz,w explicitly. 

Recall that the Hilbert space H{Tz,w) is the inductive limit of a chain (4.1) 

and that the vector ^ belongs to all spaces H^, which carry representations _Rjv- 
Evidently, for each iV, ^ is a diag(C/(A^))-invariant cyclic vector in the biregular 
representation Rj^. The pair {Rpf,£^) gives rise to a spectral measure Pz^ on 

U{N) = SGN{N). Since SGN(A'') is a discrete space, this is a purely atomic 

probability measure. It has a very simple meaning. According to decomposition 
(2.1) we obtain an orthogonal decomposition of ^ into a sum of certain vectors 
We have 

l = lia'= E ll^^f and P^Z\X) = Uxr for A G SGN(iV). 
AeSGN(Ar) 

The numbers P^%\x) can be computed, the result is as follows 

N 

pW(A) = constAr-nW'jv(Ai-i)- [] {Xi-Xj-i+jf, (4.2) 

i=l l<i<j<N 

WN{l) = \T{z-l)r{w + N+l + l)\-^ , leZ, (4.3) 

where const jv is a normalization constant. The assumption that z, w are not integers 
just means that Pz^(X) does not vanish (which is related to cyclicity of vector ^). 
The assumption ^{z + w) > —1 guarantees that 

N 

E n^^(^^-*) n {Xi-Xj-i+jf <oo 

A6SGN(Ar) i=l l<i<j<N 

for all A^, so that the normalization is indeed possible. 
On the other hand, one can prove that 

lim (P^^J) = Pz,^ , (4.4) 

where the cmbcddings ipf : SGN(A^) il were specified in Definition 3.4. Thus, 
Problem 4.3 admits a reformulation which already has a very concrete form: 

Problem 4.4. Compute explicitly the limit probability measure in the right-hand 
side of (4.3), where the probability measures in the left-hand side are given by (4.2) 
and Definition 3.4- 

In the remaining part of the paper we explain how this problem is solved. A 
detailed exposition of the material of this section can be found in Olshanski [017]. 
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5. Random point processes 



The spectral measures Pz,w that we aim to describe hve on a "very big" space 
fl, which is a domain in an infinite-dimensional product space. There is no hope 
that fl possesses a simple reference measure (like Lebesgue measure) such that Pz,w 
would be determined by a density with respect to that measure. Thus, we have to 
use another language to describe our measures. It turns out that such a language 
is provided by the theory of random point processes. 

In this section wc give a few necessary basic definitions concerning random point 
processes and also provide a few examples which seem to be relevant for the dis- 
cussion of our main problem. One should not regard our exposition as a survey on 
point processes. As basic references on this subject the reader can consult Daley 
and Vere- Jones [DVJ] and Lenard [Len] 

Let X be a locally compact space. A point configuration in X is a finite or count- 
able subset without limit points. Let Conf(X) be the set of all point configurations. 
For any Borel subset A C X with compact closure, let A/a : Conf(X) Z+ be 
the function defined by MAiX) = \Ar\ X\, where X e Conf(X). Consider the 
sigma algebra of subsets in Conf(X) generated by all functions Ma- A probability 
measure V defined on this sigma-algebra is called a random point process on X. 
Given V, point configurations X c X become random objects, and we can speak, 
for instance, about probabilities of events like this: 

AfAriX) = ni,...,MAk{X) =nfc. 

Example 5.1 (Poisson process). The simplest and most known random point 
process is the Poisson process, which is determined by an arbitrary measure m on 
X. The Poisson process is characterized by the property that the probability of 
each event of the form above, where Ai,...,A}~ do not intersect, equals 

A ^^(A,) (to(a,))"' 



Given a point process V on X, we can integrate various functions F{X) on 
Conf (X). An important class of functions F is defined as follows. Let f{xi, . . . , a;„) 
be a continuous function on X" with compact support; we set 

Ff{X)= f{xu...,xn), XeConf(X), 

summed over all n-tuplcs of pairwise distinct points in X. Note that Ff depends 
on the symmetric part of / only. Under mild assumptions on V, there exists a 
unique symmetric measure p„ on X" such that for any / as above, 

Ff{X)V{dX) = j f{xi,...Xn)Pn{dxi...dXn), 

Conf(:X) 3^" 

and, moreover, V is uniquely determined by the infinite sequence of measures 
pi,P2, • • • (see Lenard [Len]). These measures are called the correlation measures 
of V. They are a convenient tool for identifying and studying a point process. 
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When V is the Poisson process, we simply have p„ = m®". For non Poisson 
processes V, the correlation measures can have a more sophisticated structure. 

In practice one can usually choose a natural reference measure m on X such that 
Pn has a density with respect to m®" for each n. Then this density is called the 
nth correlation function oiP; we will denote it as . . . , If X is a discrete 

space and m is the counting measure then pn{xi , . . . , Xn) is the probability that 
the random configuration X contains all points xi, . . . ,Xn (if these points arc not 
all distinct then Pn{xi, . . . ,Xn) = 0). When X is not discrete, pn{xi, ■ ■ . ,Xn) can 
be informally defined as follows 

, , Prob{ random X intersects Axi, . . . , Aa;„} 

pn{xi, ...,x„)= hm — — — — , 

Axi^O,...,Ax„^0 m(Axi) . . . m{Axn) 

where Axi, . . . , Aa;„ are small neighborhoods of the points x\,. . . ,Xn- In words, 

Pn{xi, . . . , Xn) is the density of the probability to find a point of the random con- 
figuration in each of n infinitesimally small neighborhoods about a;i, . . . ,a;„. 

Definition 5.2 (Determinantal processes). Assume that a reference measure 
as above exists, so that we can deal with the correlation functions. Then V is called 
a determinantal point process if there exists a function K{x,y) on 3C x 3C such that 

Pn{xi, . . . ,Xn) = det[K{Xi,Xj)]i<ij<n , U = 1,2, . . . . 

We call K the correlation kernel of V. 



If K is symmetric {K{x, y) = K{y, x)) then the points in the random configura- 
tion arc negatively correlated: a very close rapprochement of points has a relatively 
small probability. So, the points look as mutually repelling particles. In a Poisson 
process, on the contrary, the points are not correlated at all; they look as nonin- 
teracting particles. A good survey on determinantal point processes is Soshnikov 
[So]. 

All the information about a determinantal process V is hidden in its correlation 
kernel K{x,y). In this respect, determinantal point processes can be compared to 
Gaussian measures where all the information is contained in the covariation matrix. 
Knowing K{x, y) we can, in principle, compute the probabilities of various natural 
events associated to V. We state the simplest but important example: 

Proposition 5.3. Let V he a determinantal point process with a correlation kernel 
K. The probability of having no particles in a region I C X is equal to the Fredholm 
determinant det(l — Kj), where Kj is the restriction of K to I x I. 

It often happens that such gap probability can be expressed through a solution 
of a (second order nonlinear ordinary differential) Painleve equation, see Example 
6.2 below. 

The most known example of a determinantal process is 
Example 5.4 (Sine process). The sine kernel is given by 

sin(7r(a; — w)) ^ 

K{x,y) = ^-^ P, x,yGR 

tt{x- y) 



(here the reference measure m is Lebesgue measure). The sine kernel determines a 
remarkable translation invariant point process on 3£ = R. 
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It is instructive to compare the sine process with the standard Poisson process 
on R (where m is again Lebesgue measure). Both processes are translation invari- 
ant, and for both processes the mean distance between adjacent points equals 1. 
However, as can be seen from computer simulations, the sample random configura- 
tions of the Poisson process are more chaotic. For the Poisson process, the distance 
between adjacent points is a very simple random variable (it has exponential dis- 
tribution), while for the sine process the corresponding distribution is expressed 
through a Painleve transcendent.^. 

For a large number of concrete examples of determinantal processes the space X 
is a subset of M, C, or Z, and the correlation kernel has the form 



^ Pi^)Qiy)-Qi^ (5.1) 

x-y 



or, more generally. 



E F,ix)G,{y) k 
K(x,y) = — , where V F,(a;)Gi(a;) = 0. (5.2) 

Such kernel are called integrable, see Its-Izergin-Korepin-Slavnov [IIKS], Deift [De], 
Borodin [B2]. 

Example 5.4 (Orthogonal polynomial ensembles). Let W{x) be a weight 
function (defined, say, on a subset X C M) and let po = l,pi,p2, ■ ■ ■ be the associ- 
ated family of orthogonal polynomials. For an arbitrary N = 1.2.... consider the 
orthogonal projection operator in L^{X,dx) ^ onto the iV-dimensional subspace 
spanned by functions pi{x)W's {x), < i < N — 1, and let K^{x,y) stand for the 
kernel of this operator. This kernel can be written in integrable form (5.1) with 

P{x) = const pn{x)W^{x), Q{x) = const pn-i{x)W^{x). 

In other words, K]^ {x, y) is equal to the classical Christoffel-Darboux kernel times 
W2{x)W2{y). The kernel K'^{x,y) gives rise to random iV point configurations 
in X. Namely, the density of probability ^° of a given configuration has the form 

N 

P(a;i,...,a;Ar) = const ]JW^(a;i) {xi-Xjf. (5.3) 

i=l l<i<j<N 

The random point processes of this type are called orthogonal polynomial ensembles. 
Note that (5.3) can be written in the Gibbsian form which is common in statistical 
physics: 



'P{xi, . . . ,xn) = const exp | — ^^logF ^{xi) — 2 ^^log \xi 

i i<j 



*This result was originally proved in Jimbo-Miwa- M6ri-Sato[JMMS], and a number of other 
proofs and extensions were later given by different authors, see Borodin-Deift [BD] for references. 
®If 3£ is a discrete set then Lebesgue measure dx is replaced by the counting measure. 
^"If the space X is discrete then one can simply speak about the probability of (xi, . . . , xn)- 
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The terms \ogV^^(x) and 21og|.Ti — Xj\^^ are interpreted as the one particle po- 
tential and the pair potential, respectively, and the whole ensemble is interpreted 
as an A''-particle log-gas system (Forrester [Fo]). 

A variety of random point processes comes from spectra of random matrices. A 
basic example is the Gaussian Unitary Ensemble (GUE) formed hy NxN Hermitian 
matrices distributed according to a Gaussian measure invariant under conjugation 
by unitary matrices from U {N) . The spectrum of such a random matrix is a random 
TV -point configuration in X = M arising from the Hermite orthogonal polynomial 
ensemble (in the notation of Example 5.4, W{x) = , the weight function of the 
Hermite polynomials). Prom other ensembles of random matrices one can also ob- 
tain the Laguerre and Jacobi orthogonal polynomial ensembles (see, e.g., Forrester 
[Fo]). 

One of the fundamental problems in random matrix theory is to study the as- 
ymptotic behavior of random matrices as their size goes to infinity. This leads, 
in particular, to studying the scaling limits of orthogonal polynomial ensembles in 
various regimes. For instance, if we focus at the A''-point Hermite polynomial en- 
semble with large A'^ in a neighborhood of the origin and scale the space variable x 
so that the mean distance between adjacent points becomes approximately 1 (which 
is achieved by the change of variable x — > x' = \/2Nx/-k), then we obtain in the 
limit N ^ oo the sine process. 

Orthogonal polynomial ensembles with discrete state space X arise in a num- 
ber of probabilistic models which include random tilings (Johansson [Jo3]) and 
directed percolation (Johansson [Jol], [Jo2]). Classical discrete orthogonal poly- 
nomials known as Charlier, Krawtchouk, Meixner, and Hahn polynomials arise in 
this fashion. 

6. Point processes Vz,w The main result 

Now we return to the spectral measures Pz,w We will explain how to convert 
them into random point processes 'Px,^ on the space 

X = M\{±i} 

(the real line with two punctures, at ^ and —\)- 
We define a projection Q. — > Conf(X) by 

a; = (a+,/?+,a-,/?-,<5+,(5-) 

^X = {at + \}U{\- /3+} U {-aT - 1} u {-i + I3J}, (6.1) 

where we omit possible O's among ,^^^01^ , , and also omit possible 1 's among 
Pt or /3~ . Note that X is bounded in M and its points may accumulate only near 
the punctures 5 and —5. 

By definition, Vz,w is the push-forward of the measure Pz,w under the projection 

f2->Conf(X). 

The projection is not injective, so that we can, in principle, loose a part of 
information about our measure Pz^w under the passage Pz,w — * 'Pz,w However, one 
can present arguments showing that the losses (if any) are negligible, sec the end of 
§9 in Borodin-Olshanski [B06]. Thus, we can regard Pz,w as a substitute of Pz,w 

The next result provides a description of the point process Vz,w and can be 
viewed as a solution of Problem 4.4. 
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Theorem 6.1 (Main result). Vz.w is a determina,nt,al point processes. Its corre- 
lation kernel can he written in integrable form (5.2) with k = 2, where the functions 
Fi, F^, G\, and G2 can be explicitly expressed through the Gauss hypergeometric 
function. 

For instance, if x > ^ and y > \ then the kernel can he written in form (5.1) 
with 

P{x) = const Ix - - J \x+ - j 

X 2F1 (z + iB, z + iB; z + z + w + w + 1; - x) j, 



Q{x) = const \ 2) (^"'"2 



X 2Fi(^z + w + l, z + w + 1; z + z + w + iB + 2; x) ^) 



Here 2Fi(a,b: c; () is the Gauss hypergeometric function with parameters a,b,c 
and argument ^. Note that this function is well defined for ^ < 0. 

We call the kernel of Theorem 6.1 the (continuous) hypergeometric kernel; let us 
denote it by K^^P'^'^s°°'^{x,y). Precise formulas for the kernel and the proof of the 
theorem are given in our paper [B06]. 

Note that the kernel K^^P'^'^^^°'^{x,y) is real valued but not symmetric. It has 
the following symmetry property instead: 

or outside (-i,i); (6.2) 
_ jj^hypergeom^y^ x) Otherwise. 

In other words, K^^^'^'^^'^"™ {x , y) is symmetric with respect to the indefinite inner 
product of functions on X given by 

[f,9]= j f{x)g{x)dx - j f{x)g{x)dx 

An explanation of this fact will be given in Remark 7.2 below. 

Since all the information about the point process Vz.w is hidden in the kernel 
j^hypergeom^^^y^^ a natural question is: What can be extracted from the explicit 
expression for the kernel? For instance, each of parameters af , f3f can be viewed 
as a random variable defined on the probability space {Q.,Pz,w)\ what can be said 
about their distribution? Here are two examples. 

The first example concerns the distribution of . The same result holds for 
aj"; it suffices to interchange z and w. 

Example 6.2 (Painleve VI). By virtue of Proposition 5.3, the probability dis- 
tribution of is given by 



Probla^" <u] = det(l - Ki^^), u > 0, 
13 



where we abbreviate 



V- — Z^'hypergeom I o ^ 1 

— l(s,+oo)x(s,+oo) ' * ^ 2 • 

Set 

z+z+w+w z — z + w — w z — z — w + w 

^1 = ^ , ^^3 = ^ , = ^ , 

/2 IN rflndet(l - /Cg) 2 , ^^31^4 

Then cr(s) satisfies the differential equation 
-a' {{s^ - i) a'f = (2 (sa' -a) a'- vlv^v^f - {a' + ulf{cj' + 4){a' + ul). 

This differential equation is the so-called ci-form of the Painleve VI equation. The 
proof can be found in Borodin-Deift [BD]. Wo refer to the introduction of that 
paper for a brief historical introduction and references on this subject. 

Our second example concerns the asymptotic behavior of parameters af , (if as 
i oo. 

Example 6.3 (Law of large numbers). We conjecture that with probability 1, 
hm (a+)i/'= = lim {(3+^^ = q{z), lim (a^)^/'= = lim {fi^f/'^ = q{w), 

fc— ^OO K—*00 fc— ^OO K—*OG 

where 

TT sin(7r(2; — z)) 



q{z) = exp - ^ 1^; - n| ^ = exp ( - 

V nGZ / ^ 



{z — z) sin(7r2;) sin(7ri;) 



This conjecture is based on the results of Borodin-Olshanski [BOl] and [B07]. The 
result should be obtained by analogy with Theorem 5.1 of [BOl]. However, we did 
not verify the details yet. 

7. Lattice approximation to process Vz^w 

Our proof of Theorem 6.1 is based on the limit relation (4.4). In §6, we have 

interpreted its right-hand side as a point process. Here we explain how to do the 
same for the left-hand side and thus to translate this relation into the language of 
random point processes. 

Comparing (4.2)-(4.3) with (5.3) we see that the measure on SGN(A'') gives 
rise to a discrete orthogonal polynomial ensemble on Z with weight function (4.3). 
Here we have used the bijective correspondence between diagrams A e SGN(A^) 
and A''-point configurations (Zi > • • • > ^at) on Z determined by relation k = A, — i. 

Since the weight Wn{1) from (4.3) has a slow (polynomial) decay at infinity, 

vFiv(o~ir'*'^'+"''"''^, /^±oo, 

it admits only finitely many orthogonal polynomials. However, due to the as- 
sumption ^{z + w) > —1, we have enough polynomials to define the orthogonal 
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polynomial cnscniblc for any N . Wc; call it the Askey Lesky ensemble, because the 
orthogonal polynomials in question were computed in Askey [As] and Lesky [Lesl] , 
[Les2]. The Askey-Lesky polynomials are relatives of the classical Hahn polyno- 
mials; they are expressed through the value of the hypcrgcomctric series 3_f2 at 
1. From the explicit expression of these polynomials we obtain the corresponding 
correlation kernel K^^^'^^'^'^^^^ {x , y) . The Askey-Lesky ensemble is an interesting 
example of a discrete log-gas system (the particles are confined to a lattice). 

However, the Askey-Lesky ensemble is only an intermediate object, we need 
to transform it further in order to visualize the modified Probenius coordinates of 
Young diagrams (see Definitions 3.3 and 3.4). 

The first step is rather simple, we shift the configuration (Zi, . . . , Zjv) by ^^^f^, so 
that the resulting correspondence between signatures and AT-point configurations 
takes a more symmetric form 

X ^ £ = {Ai + ^^Y^, A2 + ^^f^, • • • , Aat-i — -^^^2^, Xn — ^^Y^}- (7-1) 

The configuration £ lives on the lattice 

Z, if N is odd; 

1 
2 



2 I Z + i if TV is even. 



The next step is less obvious. Let us divide the lattice X*^^' into two parts, which 
will be denoted by and x'^^: 

v(^) _ f N-l N-3 N-3 N-1 ] 

•^in I 2' 2''''' 2' 2/' 

'rW _ ( Af+3 N+1 1 I I f N+1 N+3 \ 

-*-out — \---) 2 ' 2 / I 2 ' 2 >•••/• 

Here , the "inner" part, consists of N points of the lattice that lie on the interval 

(~T' t)' w^il^ ^out^ the "outer" part, is its complement in consisting of the 

points outside this interval. 

Given an iV-point configuration £ on X'^-*, which we interpret as a system 
of particles occupying A'' positions on the lattice we assign to it another 

configuration, X, formed by the particles in x'^^ and the holes (i.e., the unoccupied 
positions) in X^^'' . Note that X is a finite configuration, too. Since the "interior" 
part consists of exactly N points, we see that in X, there are equally many particles 
and holes. However, their number is no longer fixed, it varies between and 2N, 
depending on the mutual location of £ and For instance, if these two sets 

coincide then X is the empty configuration, and if they do not intersect then |X| = 
2N. 

We call the procedure of passage £ 1— > X the particles/holes involution. Under 
this procedure, our initial random iV-particle system (coming from the Askey- 
Lesky ensemble) turns into a random system of particles and holes. Note that the 
map £ X is reversible, so that both random point processes are equivalent. Let 
us denote the second point process by . 

The significance of the procedure described above becomes clear from the fol- 
lowing combinatorial fact. 
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Lemma 7.1 ([B06, §4]). Let A € SGN(A^) be a signature, C C X^^) be the N 
particle configuration defined by (7.1), and X c X'^^^ be the corresponding finite 
configuration of particles and holes as defined above. Let also af and bf be the 
modified Frobenius coordinates of the Young diagrams A^, see Definitions 3.3 and 
3.4. 

Then we have 

XnxW = {a+ + ^}U{-a--f}, XnxW={f-6+}U{-f +67}. (7.2) 



Comparing (7.2) with (6.1) suggests that if we shrink our phase space X'^''^^ by 

^ turn into ±^) 



the factor of N (so that the points zt^ turn into zt^) then our discrete point 



process vi%^ should have a well-defined scaling limit. We prove that such a limit 
does exist and it coincides with the point process 'Pz,w on X = M \ {±^} as defined 
in §6. 

The discrete process is determinantal, and its correlation kernel can be 

obtained by a transformation of the kernel K^^^'^^'^^^^^ {x, y); let us denote this new 
kernel by K^^^^'^"''^^ {x, y). The correlation kernel /fhypergeom^^.^ of Theorem 6.1 
is obtained as a scaling limit of the kernel K'^^^'^^'^'^'^'^^ {x , y) . 

We just gave a rough sketch of the proof of Theorem 6.1. The detailed proof 
(see Borodin-Olshanski [B06]) is rather long and technical. The main technical 
difficulties arise when we want to get a convenient explicit expression for the kernel 
^Askey-Lesky^^^ when at Icast One of variables x,y is in the "interior" 

part of the lattice. Here we apply a discrete version of the formalism of the 
Riemann-Hilbert problem, see Borodin [B2]. 

Remark 7.2 (On symmetry (6.2)). Now we arc in a position to explain the 
indefinite-type symmetry (6.2); the same kind of symmetry occurs already in the 
kernel K^^'^^'^'^^^^ (x , y). It turns out that the particles/holes involution just con- 
verts the usual symmetry of kernel K^^^^^'^^^^^ {x , y) into the indefinite-type sym- 
metry of kernel K^^^'^^'^^^^^ {x , y) . 

The point process Vi'^JJ can be viewed as a discrete two -component log gas sys- 
tem consisting of oppositely signed charges. Systems of such a type were earlier 
investigated in the mathematical physics literature (see, e.g., a number of references 
listed in section (f) of the introduction to Borodin Olshanski [B06]). However, the 
known concrete models arc quite different from our system. 

Remark 7.3 (Limit density). Given an A^-point orthogonal polynomial ensem- 
ble, let us attach to a configuration {xi, . . . , xn} a probability measure, 

jf{S^^ H +S^^). 

Under an appropriate scaling limit as A'^ — » 00, this random measure can converge to 
a (nonrandom) probability measure describing the global limit density of particles. 
For instance, in case of QUE, the limit density is given by the famous Wigner's 
semi-circle law, see e.g., Forrester [Fo, ch. 1]. 



^^This part of the kernel describes the correlations of holes with particles and other holes. The 
correlations involving particles only are described by the kernel K^"^"^'^"^^-^ {x,y) restricted to 
the "exterior" part of the lattice. 
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When wo apply this procedure to the Askey Lesky ensemble (or rather to its 
shift by ^^^) then it can be shown that, as N gets large, almost all N particles 
occupy positions inside (— y)' (R-scall that there are exactly N lattice points in 
this interval, hence, almost all of them are occupied by particles.) In other words, 
this means that the density of our discrete log-gas is asymptotically equal to the 
characteristic function of the A/'-point set of lattice points inside {—^,^-), so that 
in the scaling limit we get the characteristic function of (— 5, ^). 

It can also be shown that after the passage £ ^ X, all but finitely many parti- 
cles/holes in X concentrate, for large N, near the points ± This explains why the 
random system of paricles/holes X converges to a limit point process (as opposed 
to the Askey-Lesky ensemble). 

8. Connection with previous work 

Let us briefly discuss two similar problems which also lead to spectral measures 
on infinite-dimensional spaces. 

The first problem was initially formulated in Kerov-Olshanski-Vershik [KOVl] . 
It consists in decomposing certain natural (generalized regular) unitary represen- 
tations Tz of the group S{oo) x S{oo), depending on a complex parameter z. In 
[KOVl], [K0V2] the problem was solved in the case when the parameter z takes 
integral values (then the spectral measure has a finite-dimensional support). The 
general case presents more difiiculties and we studied it in a cycle of papers (see 
the surveys Borodin-Olshanski [B02], Olshanski [016] and references therein). Our 
main result is that the spectral measure governing the decomposition of can be 
described in terms of a determinantal point process on the real line with one punc- 
tured point. The correlation kernel was explicitly computed, it has integrable form 
(5.2), where k = 2 and the fmictions Fi, F2, Gi, and G2 are expressed through 
a confiuent hypergeometric function (specifically, through the W-Whittaker func- 
tion), see Borodin [Bl], Borodin-Olshanski [B03]. 

The second problem deals with decomposition of a family of unitarily invari- 
ant probability measures on the space of all infinite Hermitian matrices on ergodic 
components. The measures depend on one complex parameter; within a transfor- 
mation of the underlying space, they coincide with the measures /i^*-* mentioned 
in the beginning of §4. The problem of decomposition on ergodic components can 
be also viewed as a problem of harmonic analysis on an infinite-dimensional Car- 
tan motion group. The main result states that the spectral measures in this case 
can be interpreted as determinantal point processes on the real line with an inte- 
grable correlation kernel of type (5.1), where the functions P and Q are expressed 
through another confluent hypergeometric function, the M-Whittaker function, see 
Borodin-Olshanski [B05]. 

These two problems and the problem that we deal with in this paper have many 
similarities but the latter problem is, in a certain sense, more general comparing 
to both problems described above. The Askey-Lesky kernel of §7 can be viewed 
as the top of a hierarchy of (discrete and continuous) integrable kernels: this looks 
very much like the hierarchy of the classical special functions. A description of the 
"S'(oo)-part" of the hierarchy can be found in Borodin-Olshanski [B04]. 
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